Generating Reproducible Statistical Analyses and Evaluation Reports: Principles, Practices, and Free Software Tools

Demonstration at Evaluation 2024: Amplifying and Empowering Voices, annual conference of the American Evaluation Association, Portland, OR

Steven J. Pierce

pierces1@msu.edu

Center for Statistical Training and Consulting, Michigan State University

2024-10-24

Outline

What do you mean by reproducible?
Why should we aim for reproducibility?
How do we achieve reproducibility?
Demonstration

What do you mean by
reproducible?

Let’s define some concepts.

Reproducible Research (RR)

… is achieved when investigators share all the materials required to exactly recreate the findings so that others can verify them or conduct alternative analyses.

Statistical Results Can Be:

Repeatable

Original analyst
Original data

Reproducible

New analyst
Original data

Replicable

New analyst
New data

RR is a Product

RR is a product of how we work, not which methods we use.

Why aim for
reproducibility?

Several forces are promoting and enabling the push toward reproducibility.

Methods Matter!

Irreproducible < Reproducible < Replicated

Important

Reproducibility is an attainable minimum standard for science^[1].

Studies that use good scientific practices yield more trustworthy, credible findings. >

At the low end of this continuum, irreproducible findings rely on investigator reputation to establish credibility because key methodological details aren’t published and the findings can’t be independently verified. This limits our ability to judge their quality and provides a poor foundation for future replication studies.

Reproducible findings can be independently verified from the original data. We can better judge study quality on its own merits instead of by investigator reputation. Furthermore, it provides methodological details that lay a solid foundation for future replications.

Finally, replicated research has the highest level of credibility because independent investigators have replicated the findings with new studies, indicating that the probability of opportunistic biases is low and the findings generalize beyond the original sample. This dramatically increases trustworthiness of a finding. >

Roger Peng has argued that RR is an attainable minimum standard for science (Peng et al., 2006).

Guiding Principles for Evaluators

Pursuing reproducibility enacts our guiding principles^[2]:

AEA logo.

Systematic inquiry
Competence
Integrity

Funders Value RR

Data sharing and reproducibility initiatives

NIH logo. NSF logo.

IES logo.

Emerging Scientific Norms

Open science:
- https://www.cos.io/open-science
- https://open.science.gov
Pre-registered methods^[3–5]
Reporting guidelines^[6–9]
Publishing raw data^[10,11]
Journal & publisher policies^[4,10–13]
Institutional policies

Publishing Technology

Career Benefits of RR

Motivation to focus on quality
Become more efficient
Create more products
Easier to get published
Get cited more often
Build your reputation

How do we achieve reproducibility?

Understand the criteria, then apply principles, practices, & tools.

Criteria for Reproducbility
^[1]

Data
Methods
Documentation
Distribution

Materials Required to Recreate Findings

Materials	Findings
Manuals & procedures	Statistics
Instruments & scoring rules	Coefficients & p-values
Codebooks	Confidence intervals
Methods applied	Effect sizes
Data mgt decisions	Model fit indices
Data files	Figures
Software & analysis scripts	Tables

Principles for Achieving Reproducibility

Collaboration
Organization
Automation

Preservation
Integration
Separation

Understanding a few guiding principles helps you figure out how to enhance reproducibility. Thinking about reproducibility as collaboration encourages us to prepare and share materials that will make our work accessible to others. Organization simplifies communicating to others how to reproduce our work. Automation helps us achieve efficiency and accuracy because scripts can be faster and more reliable than manual processes. The preservation principle guides us to keep our raw data pristine and to keep a good audit trail. The integration principle suggests building documentation directly into data files, putting comments in code, and using code to directly insert statistical results into reports. The separation principle helps us manage complexity by splitting projects or tasks into pieces; for example separating data management from analysis, or keeping raw data separate from working data files used for specific analyses. Keep an eye out for instances of these principles in action throughout the rest of this talk.

Workflow Woes

Use GUI, menus, & dialog boxes to do tasks
Manually updating data files
Version control through saving to new file names
Disorganized folders & files
No audit trail
Copy & paste output from stats software to word processor
Fixing mistakes took lots of time

Here are some common workflow woes that may sound uncomfortably familiar from your own experience. As a student, I was taught a manual workflow for doing data analysis and writing reports. We used the GUI, menus, and dialog boxes in SPSS to do data management and analysis. We updated data files manually. We created tons of files, saving to a new filename as changes accrued. That led to confusing proliferation of folders and files with little planned structure. It was hard to tell who had last edited various files and which versions to use. We retyped or pasted results from SPSS output into Word documents then spent lots of time formatting them. If we made a mistake, it could takes hours or even days to re-do things. Over time I learned some strategies to improve that workflow, such as using SPSS syntax files to automate tasks, adding comments in the syntax files, adopting file naming conventions, and so on. But part of the problem was that the software tools weren’t integrated and lacked support for better workflows. Technology has vastly improved since my student days, but many evaluators still use manual workflows like those I just described. They don’t know what is possible now. Given what I’ve learned in recent years, if asked to resume using those older tools and workflows, I’d probably break out my favorite Spock quote.

Spock

“Captain, you’re asking me to work with equipment which is hardly very far ahead of stone knives and bearskins.”

Star Trek (1966) - S01E28 The City on the Edge of Forever

Free, Open Source Software Tools for Reproducible Research

R^[14], for statistical computing
RStudio Desktop^[15], an editor
Quarto^[16], for dynamic documents
Git^[17], for version control
GitHub.com, for sharing & collaborating on code

Dynamic Documents Via Quarto + R

Dynamic documents are a crucial technology for reproducibility. They implement the concept of literate programming, where one embeds code directly in a document such as a report, interleaving it with narrative text, headings, and in-text citations. When you render dynamic documents, the code is executed and the results are inserted into the output. That includes statistics computed from the data, figures, tables, formatted equations, fully-formatted reference sections, and more. This diagram shows the files involved for a simple example. The idea is that rendering your report script produces an output such as a PDF file that is ready for delivery to your client.

Several principles are in action here: Separation of data from code, integration of R code into the Quarto script, where it automates data management and analyses. Quarto automates inserting results into the report. The BibTeX file contains data about references you cite in the report, while the CSL file tells Quarto how to format the citations and the reference list. I’m using similar file names to stay organized: they show which script creates the report. Saving all the files follows the preservation principle. One can rapidly reproduce the report by re-rendering the script, or fix an error in either the data or the script then render again to automate updating the report.

A Research Compendium …

organizes digital files so others can review or use them to reproduce results, or do new analyses.^[18] It should:

Use prevailing conventions to organize the project
Separate data, method, & output but show how they are related
Specify the software used in the original analysis

R Packages are …

folders of digital files designed for sharing code & help documentation.^[19]

Organize files into a conventional structure
Can contain data, meta-data, & other documentation
Can contain dynamic documents & rendered output

Tip

Use an R package for a research compendium!^[18]

RStudio Projects are …

folders of files that:

Keep files for an analysis project together
Can have subfolders for organizing files
Serve as R working directories w/ workspace & history
Store settings in an RStudio project file (*.Rproj)

Tip

Turn your compendium into an RStudio project!

Git Repositories are …

folders of files tracked by Git for version control.^[17,20,21]

Preserve a history of changes to each tracked file
Allow recovering a file’s prior state from the history
Can be local or remote (hosted on a server)
Can be private or public

Tip

Put your research compendium in a Git repository!

Git is software for doing version control on code and other files stored in repositories, which are just–you guessed it–digital folders. You can choose which files to track, then log all committed changes to them, when they were made, who made them, and why. That preserves a fantastic audit trail. You can recover the exact state of a file as it existed at any commit in the history. Repositories can be either private or public. I usually keep a repository private until the corresponding paper has been published, then make it public to enhance reproducibility. Putting your compendium in a Git repository both follows the principles I’ve highlighted and solves some of those worklflow woes I described earlier. For example, you can stop using long, confusing filenames as a crude form of version control. Give each file a simple name that stays constant then let Git track how it changes over time. Jenny Bryan’s paper on using proper version control persuaded me to start using Git. I have not regretted it! > So, put your compendium in a Git repository!

GitHub

Quarto Projects are …

folders of files that allow:

Rendering some or all dynamic documents
Sharing a configuration file across documents
Redirecting output to another folder
Freezing rendered output unless the code has changed

Tip

Turn a subfolder of your compendium into a Quarto project!

Example Folder Structure

MyStudy/            [Compendium, Git repository, R package, RStudio project]
  - .git/           [Hidden folder, holds Git tracking database]
  - .Rproj.user/    [Hidden folder, holds RStudio temporary files]
  - data/           [Holds R data files created by scripts]
  - man/            [Holds R help files for package & custom functions]
  - R/              [Holds R scripts w/ custom functions]
  - scripts/        [Quarto project, holds dynamic documents]  
    - extdata/      [Holds external data files to be imported]
    - output/       [Holds rendered output]
  - .gitignore      [Tells Git what to omit from tracking]
  - DESCRIPTION     [R package meta-data]
  - MyProject.Rproj [RStudio project file & settings]
  - NEWS.md         [News for users re: changes to package]
  - README.Rmd      [Dynamic document, creates README.md]
  - README.md       [Rendered output, R package documentation]

Suppose I’ve got a project named MyStudy. Here’s what its folder structure might look like. The MyStudy/ folder is a research compendium that is also a Git repository, an R package, and an RStudio project. It is organized with several subfolders that help separate R data files from external data in other software formats that must be imported before use. It also separates data from code for either custom functions (the R subfolder) or for dynamic documents that are in a scripts subfolder that is also a Quarto project. Much of this structure is driven by R package conventions. Notice that output is also separated from the code by tucking it in its own subfolder. There are a handful of files for storing package meta-data and documentation, controlling what Git will track and what it will ignore. This basic structure has worked very well for me across several projects.

Collaboration Stakeholders

Your future self
Your team
Peer reviewers
Your audience

Organization Tips

Add documentation!
Use file naming conventions to links scripts & output
Use data flow diagrams to show relations between files
Use a primary rendering script to run other scripts

Example Data Flow Diagram

Automation Tips

Use scripts instead of GUIs
Write self-contained scripts
Render entire script frequently
Computers are patient
Computers are idiots

For automation, I suggest using scripts to do as much of the work as possible. Manual steps reduce the efficiency and reproducibility of your work. They are more error-prone than well-written code. Each script should start from a clean R session–an empty workspace–and use code to load the data it needs to run. You also want to render each script in its entirety frequently as you develop it. That’s the best way make sure the whole thing works correctly.

Leverage the inhuman patience of computers. They’re great at long, repetitive tasks that bore or fatigue humans into making errors. If something is mind-numbingly tedious, write code to get the computer to do it for you. Remember: computers are also literal-minded idiots. They do what you tell them to do, whether or not that makes sense or is what you intended. Put your time into giving them the right instructions and checking that you got the right results.

Preservation Tips

What should you preserve?

Data and meta-data
Methodology decisions & rationales
Scripts (data cleaning, management, & analysis)
Version histories of data, code, & output
Software environment & versions used

Integration Tips

Attach meta-data directly to data
Use Quarto to integrate narrative text & code
Add R functions & help files to your compendium
Include citations & reference lists

Separation Tips

Separate This		From That
Raw Data	\|	Cleaned Data
Code	\|	Data
Code	\|	Output
Data Management	\|	Data Analysis
Draft Outputs	\|	Final Deliverables
Current Files	\|	Version History

Quarto Scripts: Anatomy

Plain text files with names ending in .qmd
Use markdown syntax to format rendered output
YAML header contains metadata, options, & parameters
Body contains the main content

Quarto Scripts: Code

Inline code lets you insert dynamic text or numbers into sentences.
Code chunks allow you to do larger computations and insert more complex results.

MyStudy Demonstration Files

Let’s take a look at actual dynamic documents!

scripts/MyStudy_Report.qmd
scripts/output/MyStudy_Report.html

Bonus Resources

R for non-programmers: A guide for social scientists^[22]
What they forgot to teach you about R^[23]
RStudio IDE user guide
Using RStudio projects
R packages: Organize, test, document, and share your code^[19]
Happy Git and GitHub for the useR^[24]
Tutorial: Hello, Quarto

References

1. Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–789. https://doi.org/10.1093/aje/kwj093

2. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles

3. Bosnjak, M., Fiebach, C. J., Mellor, D., Mueller, S., O’Connor, D. B., Oswald, F. L., & Sokol, R. I. (2022). A template for preregistration of quantitative research in psychology: Report of the Joint Psychological Societies Preregistration Task Force. American Psychologist, 77(4), 602–615. https://doi.org/10.1037/amp0000879

4. DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015). Opportunistic biases: Their origins, effects, and an integrated solution. American Psychologist, 70(6), 499–514. https://doi.org/10.1037/a0039191

5. Moore, D. A. (2016). Preregister if you want to. American Psychologist, 71(3), 238–239. https://doi.org/10.1037/a0040195

6. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191

7. Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010). CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomized trials. Journal of Clinical Epidemiology, 63(8), e1–e37. https://doi.org/10.1016/j.jclinepi.2010.03.004

8. Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Journal of Clinical Epidemiology, 63(8), 834–840. https://doi.org/10.1016/j.jclinepi.2010.02.005

9. von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007). The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. PLoS Medicine, 4(10), e296. https://doi.org/10.1371/journal.pmed.0040296

10. Hrynaszkiewicz, I., & Altman, D. G. (2009). Towards agreement on best practice for publishing raw clinical trial data. Trials, 10(1), 17. https://doi.org/10.1186/1745-6215-10-17

11. Hrynaszkiewicz, I., Norton, M. L., Vickers, A. J., & Altman, D. G. (2010). Preparing raw clinical data for publication: Guidance for journal editors, authors, and peer reviewers. Trials, 11(1), 9. https://doi.org/10.1186/1745-6215-11-9

12. Laine, C., Goodman, S. N., Griswold, M. E., & Sox, H. C. (2007). Reproducible research: Moving toward research the public can really trust. Annals of Internal Medicine, 146(6), 450–453. https://doi.org/10.7326/0003-4819-146-6-200703200-00154

13. Peng, R. D. (2009). Reproducible research and biostatistics. Biostatistics, 10(3), 405–408. https://doi.org/10.1093/biostatistics/kxp014

14. R Development Core Team. (2024). R: A language and environment for statistical computing (Version 4.4.1) [Computer Program]. R Foundation for Statistical Computing. http://www.R-project.org.

15. RStudio Team. (2024). RStudio Desktop: Integrated development environment for R (Version 2024.04.2+764) [Computer Program]. Posit Software, PBC. https://posit.co

16. Allaire, J. J., Dervieux, C., Scheidegger, C., Teague, C., & Xie, Y. (2024). Quarto (Version 1.5.57) [Computer Program]. Posit Software, PBC. https://quarto.org

17. Torvalds, L., Hamano, J. C., & other contributors to the Git Project. (2024). Git for Windows (Version 2.47.0(1)) [Computer Program]. Software Freedom Conservancy. https://git-scm.com

18. Marwick, B., Boettiger, C., & Mullen, L. (2018). Packaging data analytical work reproducibly using r (and friends). The American Statistician, 72(1), 80–88. https://doi.org/10.1080/00031305.2017.1375986

19. Wickham, H., & Bryan, J. (2021). R packages: Organize, test, document, and share your code. O’Reilly Media. https://r-pkgs.org

20. Bryan, J. (2018). Excuse me, do you have a moment to talk about version control? The American Statistician, 72(1), 20–27. https://doi.org/10.1080/00031305.2017.1399928

21. Chacon, S., & Straub, B. (2014). Pro Git. Apress Media. https://git-scm.com/book/en/v2

22. Dauber, D. (2024). R for non-programmers: A guide for social scientists [Electronic Book]. https://r4np.com/

23. Bryan, J., Hester, J., Pileggi, S., & Aja, E. D. (2024). What they forgot to teach you about R. https://rstats.wtf

24. Bryan, J., TAs, T. S. 545., & Hester, J. (n.d.). Happy Git and GitHub for the useR [Web Page]. https://happygitwithr.com

Discussion Time

Thank you for attending!

Generating Reproducible Statistical Analyses and Evaluation Reports: Principles, Practices, and Free Software Tools

Outline

What do you mean by reproducible?

Reproducible Research (RR)

Statistical Results Can Be:

RR is a Product

Why aim for reproducibility?

Methods Matter!

Guiding Principles for Evaluators

Funders Value RR

Emerging Scientific Norms

Publishing Technology

Career Benefits of RR

How do we achieve reproducibility?

Criteria for Reproducbility [1]

Materials Required to Recreate Findings

Principles for Achieving Reproducibility

Workflow Woes

Spock

Free, Open Source Software Tools for Reproducible Research

Dynamic Documents Via Quarto + R

A Research Compendium …

R Packages are …

RStudio Projects are …

Git Repositories are …

GitHub

Quarto Projects are …

Example Folder Structure

Collaboration Stakeholders

Organization Tips

Example Data Flow Diagram

Automation Tips

Preservation Tips

Integration Tips

Separation Tips

Quarto Scripts: Anatomy

Quarto Scripts: Code

MyStudy Demonstration Files

Bonus Resources

References

Discussion Time

What do you mean by
reproducible?

Why aim for
reproducibility?

Criteria for Reproducbility
^[1]